The time-of-flight measurement principle is well known in the field of 3D imaging. 3D cameras (or range cameras) are known that acquire range images in real time based on the time-of-flight (TOF) principle. Such camera generally comprises a light source emitting sinusoidally modulated light into the scene to be imaged in 3D and a pixel array on which the scene is imaged by an optical system. The camera then correlates the light detected in the pixels with the light emitted and determines, for each pixel, the phase difference between emitted and received light. This phase difference is proportional to the radial distance between the camera and the part of the scene that is imaged onto the pixel concerned. As the demodulation is synchronously performed for all pixels of the pixel array, the camera provides an array of distance values associated each to a particular pixel and thus to a particular part of the scene. In the following, we will also use “phase” instead of “phase difference”; it is understood that the phase of the emitted light or a clock signal, used for modulating the emitted light or derived from the modulation of the emitted light, then serves as a reference phase. It should also be noted that, as used herein, “phase” and “phase difference” always refer to the phase of the modulation, not to the phase of the carrier wave that is modulated.
The demodulation process, which leads to the determination of the phase of the light impinging on the pixels, can be carried out in different ways. EP 0 792 555 discloses a 3D camera with a one- or two-dimensional pixel array, each pixel thereof comprising a light-sensitive part, in which charge carriers are generated in response to light impinging thereon, and a light-insensitive part with a plurality of electrical switches and storage cells associated with a respective one of these switches. The charges that are integrated in the light-sensitive part are transferred to the storage cells by sequential actuation of the electrical switches. The electrical switches are controlled in such a way that the charges transferred to a particular storage cell belong to a time interval or time intervals at a known phase of the emitted light. The charges accumulated in the different storage cells are then used to determine the phase of the light having impinged on the pixel, its amplitude and a background light level. More details on that principle of measurement can be found in the paper “The Lock-In CCD—Two-dimensional Synchronous Detection of Light” by Spirig et al. in IEEE Journal of Quantum Electronics 31 (1995), 1705-1708. An improvement of this method of demodulation is described in EP 1 659 418.
U.S. Pat. No. 6,825,455 discloses another way for demodulating the detected light. In this document, the light-sensitive part of each pixel comprises at least two modulation photogates and the light-insensitive region comprises accumulation gates, each of which is associated to a respective modulation photogate. During a first exposition interval, charge carriers generated in the light-sensitive part of the pixel in response to light impinging thereon are exposed to a first voltage gradient modulated at the frequency of the modulation of the emitted light and thereby caused to drift into a first accumulation gate when the voltage is of a first polarity and into a second accumulation gate when the voltage is of the opposite polarity. The charges qa and qb so accumulated in the first and second modulation gates, respectively, are determined. During a second exposition interval, charge carriers generated in the light-sensitive part of the pixel are exposed to a second voltage gradient modulated at the same frequency but shifted by a known phase with respect to the first voltage gradient. The charge carrier are again caused to drift into two different accumulation gates in accordance with the polarity of the voltage applied, giving rise to accumulated charges qc and qd. The phase of the light impinging on the pixel is determined using the values of the accumulated charges. If the phase difference between the voltage gradients amounts to 90°, the phase of the light can be determined as φ=arctan [(qc−qd)/(qa−qb)]. Above-cited documents are herewith incorporated herein by reference in their entirety.
For the sake of comprehensibility of the invention, we will briefly recall the basic mathematical concept of the measurement according to the TOF principle in a 3D camera working with continuously modulated light.
An illumination unit of the camera emits a continuously modulated light intensity that can be described by the formula:S(t)=S0·(1+sin(ωt))  (1)
where S0 is the average light intensity and ω is given by the modulation frequency f, i.e. ω=2πf. The scene is thus continuously illuminated with a light power density P that depends on the illumination strength S, the spatial distribution of the light and the distance between scene and camera. A part of the light power, given by a remission coefficient ρ, is then remitted by the objects in the scene. As used herein, “remission” designates reflection or scatter of light by a material. The imager optics maps the remitted light that passes through the optical system (e.g. comprising one or more lenses and/or prisms and/or filters, etc.) onto the pixel array of the camera. Assuming an ideal optical system, the received light intensity I(x,t) that arrives at time t on pixel position x=(u,v) of the pixel array thus has the following characteristics:
The light intensity I(x,t) is modulated in time with the same frequency as the emitted light, however, with the phase retarded by a value φ proportional to the distance r between the camera and the part of the scene that is mapped to point x=(u,v) on the imager. Mathematically, the received light intensity is thus given by the formulasI(x,t)=B(x)+A(x)sin(ωt−φ(x))  (2)andφ(x)=2r(x)ƒ/c  (3)with c denoting the speed of light, A the amplitude of the modulation of the received light and B (>A) the constant offset of the modulated light and background light originating from other light sources illuminating the scene (e.g. the sun). One assumes here that A, B and φ are at most slowly varying, so that they may be regarded as constant on the timescale of the modulation.
The amplitude A is thus proportional to the power density P on the part of scene that is mapped onto the corresponding pixel by the optical system, the remission coefficient ρ of that part of the scene and parameters of the optical system that are independent on the light power, like the F-number of the lens.
An ideal optical system maps a point in the scene onto a point in the image plane. In practice, however, light originating from a point in the scene is spread on an area around the theoretical image point. Various physical mechanisms may cause such spread of the image point. Defocusing of the lens causes a locally confined spread area that makes an image to appear unsharp. The relation between sharpness, defocusing and the spread area is described by the concept of depth of field. Other mechanisms leading to a point spread are light diffraction (in case of a small aperture of the lens), multiple light scattering on surfaces of the optical system or light scattering due to a contamination of the surface of the optical system. These physical effects lead to a loss of contrast in the image.
The effect of light spreading of a point source can be described mathematically by the so-called point spread function (PSF). If x=(u,v) and x′=(u′,v′) define two points in the image plane, the value g(x′,x) of the PSF function g indicates the relative amount of light that is mapped onto point x′ when the theoretical image point is x. Due to the superposition principle (that is valid for linear systems like an optical imaging system), the effect of the light spreading onto an image can be described by a convolutionI′=g*I  (4a)that isI′(x)=∫g(x, x′)I(x′)dx′,  (4b)where I denotes the ideal image and I′ the actual image affected by light spreading in the optical system. If an image is given as discrete points (pixels) the integral represents a sum over the pixels.
In order to reconstruct the ideal image I from an image I′ provided by the optical system, convolution (4a) or (4b) has to be inverted. This inversion is called a de-convolution and is in the ideal case realized by convolution of the image I′ with a de-convolution function g′ (which fulfils, at least approximately, the condition that it's convolution with g is the Dirac-delta function). Such a de-convolution function is, however, not known in general and also not always uniquely defined. A standard approach toward de-convolution is based on the transformation of the image and the convolution function to Fourier space. However, this approach is not always applicable. In the field of image processing various approaches have been developed for de-convoluting an image at least approximately.
In 2D imaging, light spreading is often negligible under normal conditions (using ambient light and well-focused optics). The point spread function g(x′,x) is then close to a delta peak, e.g.:g(x′,x)=(1−ε)δ(x′,x)+εƒ(x′,x)  (5)where ƒ is a function normalized to 1 and ε the relative amount of light scattered in the optical system. In a well-focused and clean optical system, ε is typically small e.g. of the order 10−3. A blur visible for the human eye therefore occurs only if light from a very bright light source (e.g. the sun) is shining into the optical system. In this case, the contribution of the spread light from the bright light source cannot be neglected, since its intensity is many orders of magnitude higher than the light intensity reflected by an object in the scene. If the optical system is contaminated (with dirt or a scratch), the parameter ε is larger, so that light spreading visible for the human eye could result even in normal lighting conditions.
The inventors have recognised that in a TOF camera system broadly illuminating the scene, an effect similar to image blurring due to scattered sun light can occur due to the active illumination. The main reason is that the light power density P on an object in the scene strongly depends on the distance d of the object to the light source (P˜1/d2). The light intensity I′(x) at the corresponding pixel position x is proportional to the light power density and the remission coefficient ρ, i.e.I′(x)˜ρ/d2  (6)
As an example, an object with a remission coefficient of 50% at a distance of 1 m will generate an intensity value, which is 1000 times larger than the intensity value generated by an object with a remission coefficient of 5% at a distance of 10 m. Therefore, when (5) is substituted into the convolution integral (4b), the contributions of intensities at points x≠x′ are no longer negligible, even if the light scattering factor ε is of the order 10−3.
It is important to recognise that the phase measurement and thus the computed distance information is falsified by light spreading. This will now be explained in more detail first for a superposition of two modulated light intensities and then for the general case.
The superposition of two modulated light intensities expressible by equation (2) yields:I′(t)=I1(t)+I2(t)=(B1+B2)+A1 sin(ωt−φ1)+A2 sin(ωt−φ2)  (7a)I′(t) can again be expressed in the form of one modulated light intensity, i.e.I′(t)=B′+A′ sin(ωt−φ′)  (7b)whereB′=B1+B2 A′=√{square root over (AS′2+AC′2)}φ′=arctan(AS′/AC′)  (8)withAS′=A1 sin φ1+A2 sin φ2=:AS1+AS2 AC′=A1 cos φ1+A2 cos φ2=:AC1+AC2.  (9)
Formulas (7a) to (9) show that the superposition of two modulated intensities with same frequency but different phases and amplitudes results in a modulated intensity with again the same frequency but whose phase depends not only on the phases but also on the amplitudes of the individual intensities being superposed. In other words, light spreading in presence of a non-ideal optical system induces errors in the measured phase values.
Before turning to the general case of superposition of modulated intensities due to spreading, it shall be observed that is convenient to rewrite equation (9) in complex notation:Â′:=A′eiφ′=A1eiφ1+A2eiφ2=:Â1+Â2  (10)where AC and AS are the real and the imaginary components, respectively, of the complex amplitude Â, i.e.AS′=ImÂ′AC′=ReÂ′  (11)
The superposition principle (9) or (10) for the amplitudes can be straightforwardly generalized for the case that the optical system spreads the light intensity with a point spread function g. UsingI(t)=B+A sin(ωt−φ)=B−Im(A·ei(φ-ωt))=B−Im(Âe−iωt)  (12)andI′(t)=B′+A′ sin(ωt−φ′)=B′−Im(A′·ei(φ′-ωt))=B′−Im(Â′e−iωt)  (13)and substituting this into equation (4b), the result isÂ′(x)=∫g(x, x′)Â(x′)dx′.  (14)
The resulting phase φ′(x) and amplitude A′(x) are again given by equation (8) using the real and imaginary parts of Â′(x) as defined in (11).
As a result of the non-negligible superposition, the contrast in phase measurement is reduced. This means that the phases measured in the different pixels are shifted towards the phase of the pixel with the strongest amplitude of modulation. This effect of phase shift is the stronger, the smaller the amplitude of the corresponding pixel is. Therefore, the phase shift caused by light spreading affects mostly background pixel. The objects in the background of the scene appear thus nearer to the camera than they actually are, especially if the background part of the scene has a low remission coefficient.
The invention generally seeks to reduce the effect of light spreading onto the range measurement.